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REVIEW OF REPORTS 

Under the Education Sciences Reform Act, all research, statistics, and evaluation reports conducted by, or 
supported through, the Institute shall be subject to rigorous peer review before being published or 
otherwise made available to the public. 

See Appendix A for relevant sections of the legislation. 

Overview 

Each of the Institute's four centers produces reports that are subject to peer review. Review of Institute 
reports occurs within the National Center generating the report and in addition, for most reports, through 
the Standards and Review Office. 



Center Review Procedures 

National Center for Education Evaluation and Regional Assistance (NCEE) 

Prior to review and approval by the NCEE Commissioner, NCEE reports are reviewed internally by the 
project officer and selected NCEE evaluation staff. In addition, members of the technical working group 
who served as advisors to the research team conducting the project typically review reports. After the 
NCEE Commissioner has approved the final draft report, the NCEE project officer submits the final draft 
report to the Standards and Review Office for external peer review. 

National Center for Education Research (NCER) 

Prior to review and approval by the NCER Commissioner, the project officer reviews NCER reports 
internally. In addition, selected NCER staff and external scientists may review reports. After the NCER 
Commissioner has approved the final draft report, the NCER project officer submits the final draft report 
to the Standards and Review Office for external peer review. 

National Center for Special Education Research (NCSER) 

Prior to review and approval by the NCSER Commissioner, the project officer reviews NCSER reports 
internally. In addition, selected NCSER staff and members of the technical working group who served as 
advisors to the research team conducting the project may review reports. After the NCSER Commissioner 
has approved the final draft report, the NCSER project officer submits the final draft report to the 
Standards and Review Office for external peer review. 

National Center for Education Statistics (NCES) 

Prior to review and approval by the NCES Commissioner, NCES reports are reviewed at 2 levels - first, by 
the program generating the report and second, by the office of the Chief Statistician for center level 
review. 

The program generating the report initially reviews NCES reports. After the program-level review, the 
Program Director and responsible Associate Commissioner approve internal NCES reviewers for the 
center-level review of the report. Occasionally the center-level review will include a reviewer outside of 
NCES. When non-NCES reviewers are used, those comments are included in the materials sent forward 
to the Standards and Review Office. In addition, the responsible Associate Commissioner reviews the 
report. 

The report is submitted to the Chief Statistician for center-wide review. The Chief Statistician assigns an 
NCES senior mathematical statistician to lead the review. At the same time, the report is forwarded to 
the leader of the technical review team for assignment to a research associate and usually 2 research 
assistants. The technical review team submits its comments on the NCES electronic review template to 
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the senior mathematical statistician assigned to the report. The senior mathematical statistician is 
responsible for reviewing the technical review team's comments and adding additional comments. 

The senior mathematical statistician sends the set of technical review comments to the NCES contact 
person for the report. The author annotates the review comments from the technical reviewers and from 
the designated internal reviewers with proposed changes to the report, or explanations as to why 
changes would not be made, and submits this to the senior mathematical statistician assigned to the 
report. 

The senior mathematical statistician reviews the proposed changes and explanations, indicates 
agreement or requests further clarification or changes, and returns the electronic template to the NCES 
contact person for changes to be made. The author submits the revised report along with the electronic 
template indicating where the changes are located in the report to the senior mathematical statistician. 
After the senior mathematical statistician approves the revised report, the report is signed off by the 
Chief Statistician and the Commissioner and sent forward to the Standards and Review Office. 



Standards and Review Office 



Categories of Review 

The Standards and Review Office receives statistics, research, and evaluation reports generated by the 
Institute's four centers. Standards and Review staff conduct two types of reviews - external scientific 
reviews and internal scientific reviews - as described below. 

The only reports that are not reviewed by the Standards and Review Office are administrative reports 
that do not include analyses of data and are not considered statistics, research, or evaluation reports. 

For example, user's manuals for datasets and methodology reports that describe research methods (e.g., 
sampling procedures) used in a study are considered administrative reports. Such reports are typically 
reviewed only in the Center generating the report. 

External Scientific Review. The Standards and Review Office sends evaluation, research, and 
statistics reports that present new analyses of data in any form other than basic cross-tabulations to 
external scientists for peer review. Review and policy documents also receive external review. 

Internal Scientific Review. Standards and Review staff review reports with limited data analyses 
internally. Such reports include evaluation, research, and statistics reports that present new analyses of 
data that are limited to basic cross-tabulations and are accompanied only by text that describes the study 
that generated the dataset and relevant information about the dataset on which the analyses were 
conducted. Also subject to internal review are indicator reports: these do not present new data analyses 
but draw from other reports or present the results of analyses in tabular form with limited text and 
virtually no information on the study (such as descriptions of samples, methods, or analyses). Examples 
of indicator reports are Indicators of School Crime and Safety: 2005 (NCES 2006-001), and Youth 
Indicators, 2005: Trends in the Well-Being of American Youth (NCES 2005-050). 

Initiation of Scientific Review Process 

Before reports are submitted to the Standards and Review Office, the reports must have met all internal 
review standards of the Center generating the report. All reports must be in final form, with tables and 
references in standard format with one exception. For most NCES and NCSER reports, descriptions of the 
types of statistical tests (e.g., t-tests) that were conducted generally appear in the appendix. 
Documentation of the actual statistical tests that were conducted is not included in the text of the 
published version of the report. When documentation of the actual statistical tests is not included in the 
text of the published version of the report, an annotated version of the report is sent to the reviewers so 
that they are able to follow the statistical tests that were conducted for the report. 
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Initiation of the review process by the Standards and Review Office begins when the relevant 
Commissioner or his or her designee sends a report to the Office of the Deputy Director for Science for 
review. The Commissioner recommends either that the document should be (1) externally reviewed, (2) 
internally reviewed, or (3) exempt from review by the Standards and Review Office because the report is 
an administrative report. If the Deputy Director for Science agrees with the Commissioner's 
recommendation and the report is to be externally or internally reviewed, the Deputy Director for Science 
assigns the report to an Action Editor in the Standards and Review Office to implement the appropriate 
review. If there is agreement that a report is exempt from review, then the Deputy Director for Science 
notifies the relevant Commissioner or designee of the decision. On those occasions when the Deputy 
Director for Science disagrees with the Commissioner's recommendation, the Deputy Director for Science 
contacts the relevant Commissioner to reach an agreement on type of review. 

Implementation of External Scientific Review 

The assigned Action Editor identifies potential scientific reviewers for the report based on the content and 
methodological expertise required for a scientific review of the report. Action Editors look for researchers 
who (1) have published research in top research journals in the content domains relevant to the study 
presented in the report, (2) have used similar methodological approaches, and (3) have substantial 
experience conducting studies similar to the one presented in the report. Action Editors submit names, 
publication lists, and examples of published work to the Deputy Director for Science for final approval of 
reviewers. 

Action Editors recruit reviewers and generally ask for reviews to be returned within 10 workdays. At least 
2 external scientific reviewers are recruited for each report. In cases in which the report covers a broad 
domain (e.g., reviews of multiple research areas) or requires demanding methodological expertise in 
addition to content expertise, 3 or more reviewers are recruited for the report. 

Reviews. Typically, reviewers are asked to consider significance of the topic for the field of 
education, quality of the scholarship, appropriateness of the conclusions, and clarity of the presentation. 
Institute reports do not include presentation of policy implications or recommendations. In addition, 
reviewers receive specific guidance on assuring that language that advances causal claims is adequately 
supported by the methods and analyses used in the report. As in journal reviews, the reviewers make a 
publication recommendation to the Action Editor indicating whether the manuscript in its present form 
should be published with minimal or no revisions, should be revised and resubmitted, or should not be 
published. 

Action Editors. Based on the Action Editor's own review of the report and the external reviewers' 
reviews, the Action Editor writes a disposition memorandum indicating whether or not the report has 
been approved or is in need of revision and sends the memorandum to the Deputy Director for Science 
for approval. 

Implementation of Internal Scientific Review 

The assigned Action Editor reviews the report considering significance of the topic for the field of 
education, quality of the scholarship, appropriateness of the conclusions, and clarity of the presentation. 
Based on this review, the Action Editor writes a disposition memorandum indicating whether or not the 
report has been approved or is in need of revision and sends the memorandum to the Deputy Director for 
Science for approval. 

Disposition Memorandum 

After completing the review of the report (external or internal) and receiving approval from the Deputy 
Director for Science, the Action Editor sends a disposition memorandum to the Institute staff person 
responsible for the report (author or project officer), copying the Commissioner or Associate 
Commissioner, the Deputy Director for Science, and other relevant staff. 
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If after reviewing the memorandum, the author has changes to propose or wishes to discuss specific 
requests, the author, or Institute project officer for reports not authored by Institute staff, communicates 
such issues to the Action Editor. After the two have agreed on changes and the revisions have been 
completed in the report, a revised report is sent to the Action Editor. After reviewing the revised report, 
the Action Editor, upon consultation with the Deputy Director for Science, may approve the report or 
request additional changes. 

The Action Editor may also indicate that the report must be cleared by members of the Institute's 
Disclosure Risk Board prior to approval for publication. 

Withdrawal of Report 

In some instances, the Center generating the report may decide that the report needs further revision 
prior to completion of the review process and may withdraw the report from consideration by notifying 
the assigned Action Editor. 

In rare instances, the review reveals a major problem in the report. In such instances, the Action Editor 
may suggest that the Center consider withdrawing the report. 

Tracking the Process 

Time is of the essence in the review process. The progress of each report is carefully tracked through 
the review, and the Standards and Review office strives to produce initial disposition memoranda within 
10 working days on average for internal reviews and within 25 working days on average for external 
reviews. 

Feedback to External Reviewers 

The external reviewers are sent copies of each other's reviews and a summary of report disposition. This 
is done without revealing reviewers' identities. 
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Legislation Relevant to Scientific Peer Review 
20 U.S.C. 9501 et. seq. 

Education Sciences Reform Act of 2002 



SECTION 102. DEFINITIONS 

(18) Scientifically based research standards . 

(A) The term "scientifically based research standards" means research standards that— 

(i) apply rigorous, systematic, and objective methodology to obtain reliable and valid 
knowledge relevant to education activities and programs; and 

(ii) present findings and make claims that are appropriate to and supported by the 
methods that have been employed. 

(B) The term includes, appropriate to the research being conducted— 

(i) employing systematic, empirical methods that draw on observation or experiment; 

(ii) involving data analyses that are adequate to support the general findings; 

(iii) relying on measurements or observational methods that provide reliable data; 

(iv) making claims of causal relationships only in random assignment experiments or other 
designs (to the extent such designs substantially eliminate plausible competing explanations for 
the obtained results); 

(v) ensuring that studies and methods are presented in sufficient detail and clarity to allow 
for replication or, at a minimum, to offer the opportunity to build systematically on the findings of 
the research; 

(vi) obtaining acceptance by a peer-reviewed journal or approval by a panel of 
independent experts through a comparably rigorous, objective, and scientific review; and 

(vii) using research designs and methods appropriate to the research question posed. 

(19) Scientifically valid education evaluation . The term "scientifically valid education evaluation" 
means an evaluation that— 

(A) adheres to the highest possible standards of quality with respect to research design and 
statistical analysis; 

(B) provides an adequate description of the programs evaluated and, to the extent possible, 
examines the relationship between program implementation and program impacts; 

(C) provides an analysis of the results achieved by the program with respect to its projected 
effects; 

(D) employs experimental designs using random assignment, when feasible, and other 
research methodologies that allow for the strongest possible causal inferences when random 
assignment is not feasible; and 

(E) may study program implementation through a combination of scientifically valid and reliable 
methods. 

(20) Scientifically valid research . The term "scientifically valid research" includes applied research, 
basic research, and field-initiated research in which the rationale, design, and interpretation are soundly 
developed in accordance with scientifically based research standards. 



IES Standards and Review Office 



Approved by NBES - January 24, 2006 




IES Procedures for Peer Review of Reports - January 24, 2006 



8 of 9 



PART A - THE INSTITUTE OF EDUCATION SCIENCES 
SECTION 114. OFFICE OF THE DIRECTOR 



(f) DUTIES . The duties of the Director shall include the following: 

(2) To ensure the methodology applied in conducting research, development, evaluation, and 
statistical analysis is consistent with the standards for such activities under this title. 

(7) To ensure that activities conducted or supported by the Institute are objective, secular, 
neutral, and nonideological and are free of partisan political influence and racial, cultural, gender, or 
regional bias. 

(g) EXPERT GUIDANCE AND ASSISTANCE . The Director may establish technical and scientific peer- 
review groups and scientific program advisory committees for research and evaluations that the 
Director determines are necessary to carry out the requirements of this title. The Director shall 
appoint such personnel, except that officers and employees of the United States shall comprise no 
more than Va of the members of any such group or committee and shall not receive additional 
compensation for their service as members of such a group or committee. The Director shall ensure 
that reviewers are highly qualified and capable to appraise education research and development 
projects. The Federal Advisory Committee Act (5 U.S.C. App.) shall not apply to a peer-review group 
or an advisory committee established under this subsection. 

SECTION 116. NATIONAL BOARD FOR EDUCATION SCIENCES. 



(b) DUTIES . The duties of the Board shall be the following: 

(3) To review and approve procedures for the technical and scientific peer review of the 
activities of the Institute. 

SECTION 120. COMPETITIVE AWARDS. 

Activities carried out under this Act through grants, contracts, or cooperative agreements, at a minimum, 
shall be awarded on a competitive basis and, when practicable, through a process of peer review. 



PART B - NATIONAL CENTER FOR EDUCATION RESEARCH 

SECTION 134. STANDARDS FOR CONDUCT AND EVALUATION OF RESEARCH. 

(a) GENERAL In carrying out this part, the Research Commissioner shall— 

(1) ensure that all research conducted under the direction of the Research Center follows 
scientifically based research standards; 

(2) develop such other standards as may be necessary to govern the conduct and evaluation of 
all research, development, and wide dissemination activities carried out by the Research Center to 
assure that such activities meet the highest standards of professional excellence; 

(3) review the procedures utilized by the National Institutes of Health, the National Science 
Foundation, and other Federal departments or agencies engaged in research and development, and 
actively solicit recommendations from research organizations and members of the general public in 
the development of the standards described in paragraph (2); and 
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(4) ensure that all research complies with Federal guidelines relating to research misconduct, 
b) PEER REVIEW . 

(1) In General. The Director shall establish a peer review system, involving highly qualified 
individuals with an in-depth knowledge of the subject to be investigated, for reviewing and evaluating 
all applications for grants and cooperative agreements that exceed $100,000, and for evaluating and 
assessing the products of research by all recipients of grants and cooperative agreements under this 
Act. 

(2) Evaluation. The Research Commissioner shall 

(A) develop the procedures to be used in evaluating applications for research grants, 
cooperative agreements, and contracts, and specify the criteria and factors (including, as 
applicable, the use of longitudinal data linking test scores, enrollment, and graduation rates 
over time) which shall be considered in making such evaluations. 

PART E - NATIONAL CENTER FOR SPECIAL EDUCATION RESEARCH 
SECTION 175. ESTABLISHMENT 

(c) APPLICABILITY OF EDUCATION SCIENCES REFORM ACT OF 2002 . Parts A and F, and the 
standards for peer review of applications and for the conduct and evaluation of research under sections 
133(a) and 134, respectively, shall apply to the Secretary, the Director, and the Commissioner in carrying 
out this part. 
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